-
-
Notifications
You must be signed in to change notification settings - Fork 41
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
util: Check if Steam configuration files exist before marking as available #356
util: Check if Steam configuration files exist before marking as available #356
Conversation
Thanks! So what
That may sound a bit oversimplified, but I wonder
That would make the constants file a bit more clean.
Yeah, might be worth adding a label. Something like |
Yup, exactly!
Huh, that's an excellent point. I wonder why it's set to a default in the first place, when I get home I'll try on
I think Python can handle them too, but I wonder if it's a good idea to rely on circular imports in this way ( It's probably a stretch but I wonder if there's any PEP guidance on this sort of thing 🤔 |
Hmm, I probably though an empty string could break something when I implemented that. Shouldn't be the case though.
Ideally I guess we would remove the logic from constants.py and completely do it in the util/main logic. Probably the easiest way would be to add something like the following below POSSIBLE_INSTALL_LOCATIONS += [
{'install_dir': f'{root}/compatibilitytools.d/', 'display_name': 'Steam', 'launcher': 'steam', 'type': 'native', 'icon': 'steam', 'vdf_dir': f'{root}/config'}
for root
in _POSSIBLE_STEAM_ROOTS
] |
EDIT: Sniped while writing this comment :-) Yep, ProtonUp-Qt/pupgui2/constants.py Lines 29 to 36 in 6cf1def
I wouldn't be against removing the conditional checks in My concern would be that the paths in The problem becomes how to deal with the duplicate paths in an efficient way. I initially thought that we could use sets to remove duplicates, but you can't convert a list of dictionaries to a set. So, one solution might be to first, store the Lines 206 to 208 in 6cf1def
realpath , and not a symlink, once we remove duplicate entries from available_dirs that should mean we only end up with unique Steam installation paths (and 99,999 out of 100,000, there would only be one non-Flatpak non-Snap Steam installation anyway).
Essentially we would end up with something like this:
We could do it like this maybe: # This part is from the snippet, untested :-)
POSSIBLE_STEAM_INSTALL_LOCATIONS = [
{'install_dir': f'{root}/compatibilitytools.d/', 'display_name': 'Steam', 'launcher': 'steam', 'type': 'native', 'icon': 'steam', 'vdf_dir': f'{root}/config'}
for root
in _POSSIBLE_STEAM_ROOTS,
]
# Steam Flatpak/Snap
POSSIBLE_STEAM_INSTALL_LOCATIONS += [
{'install_dir': '~/.var/app/com.valvesoftware.Steam/data/Steam/compatibilitytools.d/', 'display_name': 'Steam Flatpak', 'launcher': 'steam', 'type': 'flatpak', 'icon': 'steam', 'vdf_dir': '~/.var/app/com.valvesoftware.Steam/.local/share/Steam/config'},
{'install_dir': '~/snap/steam/common/.steam/root/compatibilitytools.d/', 'display_name': 'Steam Snap', 'launcher': 'steam', 'type': 'snap', 'icon': 'steam', 'vdf_dir': '~/snap/steam/common/.steam/root/config'},
]
# This is our existing `POSSIBLE_INSTALL_LOCATIONS`, just with the Steam paths moved to the above list
POSSIBLE_LAUNCHER_INSTALL_LOCATIONS = [
{'install_dir': '~/.local/share/lutris/runners/wine/', 'display_name': 'Lutris', 'launcher': 'lutris', 'type': 'native', 'icon': 'lutris', 'config_dir': '~/.config/lutris'},
{'install_dir': '~/.var/app/net.lutris.Lutris/data/lutris/runners/wine/', 'display_name': 'Lutris Flatpak', 'launcher': 'lutris', 'type': 'flatpak', 'icon': 'lutris', 'config_dir': '~/.var/app/net.lutris.Lutris/config/lutris'},
{'install_dir': '~/.config/heroic/tools/wine/', 'display_name': 'Heroic Wine', 'launcher': 'heroicwine', 'type': 'native', 'icon': 'heroic'},
{'install_dir': '~/.config/heroic/tools/proton/', 'display_name': 'Heroic Proton', 'launcher': 'heroicproton', 'type': 'native', 'icon': 'heroic'},
{'install_dir': '~/.var/app/com.heroicgameslauncher.hgl/config/heroic/tools/wine/', 'display_name': 'Heroic Wine Flatpak', 'launcher': 'heroicwine', 'type': 'flatpak', 'icon': 'heroic'},
{'install_dir': '~/.var/app/com.heroicgameslauncher.hgl/config/heroic/tools/proton/', 'display_name': 'Heroic Proton Flatpak', 'launcher': 'heroicproton', 'type': 'flatpak', 'icon': 'heroic'},
{'install_dir': '~/.local/share/bottles/runners/', 'display_name': 'Bottles', 'launcher': 'bottles', 'type': 'native', 'icon': 'com.usebottles.bottles'},
{'install_dir': '~/.var/app/com.usebottles.bottles/data/bottles/runners/', 'display_name': 'Bottles Flatpak', 'launcher': 'bottles', 'type': 'flatpak', 'icon': 'com.usebottles.bottles'}
]
# Essentially this means we split the Steam paths and all other paths into their own lists, and then store the combined list as 'POSSIBLE_INSTALL_LOCATIONS'
# Doing it this way means we keep the Steam paths at the beginning of the list
# This approach be very marginally more readable since it means we keep the Steam comprehension bits separated, but really, I don't think it's a huge deal either way :-)
POSSIBLE_INSTALL_LOCATIONS = POSSIBLE_STEAM_INSTALL_LOCATIONS + POSSIBLE_LAUNCHER_INSTALL_LOCATIONS This is all kind of off-the-top-of-my-head, I haven't sat down and tested this out much yet. It's just one approach I think could work. In short, I think your idea of storing all the install paths is a fine approach, even if it means multiple paths and possibly multiple invalid/duplicate paths for the same launcher type. It could indirectly allow for the benefit of picking up multiple "Steam roots" (multiple "regular" Steam installs). But we'd probably need to handle removing duplicates, so we don't end up listing all of the existing symlinks on top of the actual Steam root. We have a couple of approaches on how we could handle this, and it means keeping the conditional "filtering" logic isolated to a utility function. But I haven't tested how much of this approach would work or what pitfalls we might encounter. We don't store multiple of the same path for the same launcher type, so I don't know if we would run into any Unforeseen Consequences:tm:. I imagine not if we handle duplicates properly in I can do some testing on this over the weekend to see how having multiple paths in P.S. - If all else fails, just doing I think the way I ended up explaining this sounds like we might be adding a lot of complexity, but my comment essentially boils down to incorporating all possible Steam install paths in |
Changes are a bit WIP across commits, I will summarize and explain them afterwards :-) |
Okay, I was planning to make a couple of other changes, but I think they're better for a separate PR. Across d0e05ce and 6423f52, the following changes were made:
|
Something has went wrong somewhere and I pushed some bad changes, will fix... |
It looks like 6423f52 is causing the problem. I narrowed the issue down to using For some reason, using |
Hm, POSSIBLE_INSTALL_LOCATIONS = [
{'install_dir': '/home/emma/.local/share/Steam/compatibilitytools.d', 'display_name': 'Steam', 'launcher': 'steam', 'type': 'native', 'icon': 'steam', 'vdf_dir': '/home/emma/.local/share/Steam/config'},
{'install_dir': '~/.var/app/com.valvesoftware.Steam/data/Steam/compatibilitytools.d/', 'display_name': 'Steam Flatpak', 'launcher': 'steam', 'type': 'flatpak', 'icon': 'steam', 'vdf_dir': '~/.var/app/com.valvesoftware.Steam/.local/share/Steam/config'},
{'install_dir': '~/snap/steam/common/.steam/root/compatibilitytools.d/', 'display_name': 'Steam Snap', 'launcher': 'steam', 'type': 'snap', 'icon': 'steam', 'vdf_dir': '~/snap/steam/common/.steam/root/config'},
{'install_dir': '~/.local/share/lutris/runners/wine/', 'display_name': 'Lutris', 'launcher': 'lutris', 'type': 'native', 'icon': 'lutris', 'config_dir': '~/.config/lutris'},
{'install_dir': '~/.var/app/net.lutris.Lutris/data/lutris/runners/wine/', 'display_name': 'Lutris Flatpak', 'launcher': 'lutris', 'type': 'flatpak', 'icon': 'lutris', 'config_dir': '~/.var/app/net.lutris.Lutris/config/lutris'},
{'install_dir': '~/.config/heroic/tools/wine/', 'display_name': 'Heroic Wine', 'launcher': 'heroicwine', 'type': 'native', 'icon': 'heroic'},
{'install_dir': '~/.config/heroic/tools/proton/', 'display_name': 'Heroic Proton', 'launcher': 'heroicproton', 'type': 'native', 'icon': 'heroic'},
{'install_dir': '~/.var/app/com.heroicgameslauncher.hgl/config/heroic/tools/wine/', 'display_name': 'Heroic Wine Flatpak', 'launcher': 'heroicwine', 'type': 'flatpak', 'icon': 'heroic'},
{'install_dir': '~/.var/app/com.heroicgameslauncher.hgl/config/heroic/tools/proton/', 'display_name': 'Heroic Proton Flatpak', 'launcher': 'heroicproton', 'type': 'flatpak', 'icon': 'heroic'},
{'install_dir': '~/.local/share/bottles/runners/', 'display_name': 'Bottles', 'launcher': 'bottles', 'type': 'native', 'icon': 'com.usebottles.bottles'},
{'install_dir': '~/.var/app/com.usebottles.bottles/data/bottles/runners/', 'display_name': 'Bottles Flatpak', 'launcher': 'bottles', 'type': 'flatpak', 'icon': 'com.usebottles.bottles'}
] Strange issue, I'll keep investigating... d0e05ce seems to work, so it's something in 6423f52 causing the issue... |
It seems like it is |
…L_LOCATIONS Fixes a strange recusion depth crash.
af05168 goes back to using f-strings for I also removed the |
Noticed a couple of outdated and unclear comments, I will rework them a bit later |
I think with ed709d9 I'm good. If there's anything outstanding here let me know and I'll try fix this up! So now overall the changes in this PR are:
I wonder if it's worth extending our Steam check to determine if the VDF files are > 0 bytes, or some other magic-btye value, like 16 bytes? I dunno if it's too valuable. In most places we just check if a file exists, although there's no reason we couldn't create a general utility function to check if a file is valid, and as part of that check, include an optional parameter to check the size of a file. For example we could create a util function: def is_file_valid(file_path: str, min_bytes: int = 0):
return os.path.isfile(file_path) and os.path.getsize(file_path) >= min_bytes There might already be a builtin way to do this or a better way, and introducing this across the codebase would probably be a multi-PR initiative, but if we want to do this kind of check, we could introduce the function here or in a follow-up PR :-) |
Thanks for the elaborate explainaiton! Code looks good.
Strange... issue says something about ConfigParser.
Sadly Sets aren't the way to go in most cases. They are neither fast nor deterministic...
I'm glad that with Flatpak/Snap they "introduced" some standard paths.
I'm not sure how likely it is that the file exists but doesn't have any content. I wouldn't do the magic checks as they could change in the future and that would break the app. |
if os.path.exists(install_dir): | ||
# POSSIBLE_INSTALL_LOCATIONS may contain duplicate paths, so only add unique paths to available_dirs | ||
# This avoids adding symlinked Steam paths | ||
if is_valid_launcher_installation(loc) and not install_dir in available_dirs: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just wondering: With this check not install_dir in available_dirs
in place, do we still need list(dict.fromkeys(_POSSIBLE_STEAM_ROOTS))
in constants.py
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In short, no, but it should help catch any case where we might end up with duplicate paths in POSSIBLE_INSTALL_LOCATIONS
.
Most probably not for functionality at least for Steam, but for consistency, list(dict.fromkeys(_POSSIBLE_STEAM_ROOTS))
(and the earlier checks to remove non-existent paths from _POSSIBLE_STEAM_ROOTS
) should ensure that POSSIBLE_INSTALL_LOCATIONS
will only contains Steam install paths that exist on the filesystem. Whether or not they're valid is another question, and that's what our function handles here.
The check might also be good in case other install paths in future could have duplicates. This probably won't ever happen, but it just makes sure at this level that we don't list the same path twice. If we really wanted it to be stricter, we could make sure all paths in available_dirs
and install_dir
itself are always expanded to their realpath
, and also we could filter out duplicates by returning list(dict.fromkeys(available_dirs))
in this function, but I think that's overkill.
For the changes in this PR, the not install_dir in available_dirs
condition is not strictly needed, it's just an extra guard.
Also, I guess the comment here is wrong, we don't list duplicate paths.
available_dirs.append(install_dir) | ||
install_dir = config_custom_install_location().get('install_dir') | ||
if install_dir and os.path.exists(install_dir): | ||
if install_dir and os.path.exists(install_dir) and not install_dir in available_dirs: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess we could use is_valid_launcher_installation
here too.
But let's leave it as is, maybe the user has some very strange Steam setup and wants to add it like this.
Should fix #353.
All mentions to just "Steam" in this issue refer to Steam installed i.e. via package manager, not Steam Flatpak or Steam Snap :-) The problem could still apply to them I think, but I don't have them on-hand to test.
Overview
This PR changes the check in
util#available_install_directories
to be a bit stricter when checking for Steam installs. It now makes sure that the configuration files we need exist before marking a Steam installation as valid.Problem
The core of the problem is that we can mark an invalid Steam installation as available because we only check if the directories that we need exist, not if the actual files we need exist. We need
libraryfolders.vdf
andconfig.vdf
to perform various actions, but inutil#available_install_directories
, we only checkos.path.exists
. This causes the crash in #353, because all the directories exist, but the files do not.The existing check loops through all the paths in
POSSIBLE_INSTALL_LOCATIONS
, which is a list of dictionaries with the installation paths for compatibility tools being under theinstall_dir
key (this distinction is important later). However it only checks ifinstall_dir
exists. In the case of Steam installations, this means it only checks if, say,~/.local/share/Steam/compatibiltytools.d
exists. But that is not sufficient for Steam at least. This is because this path can exist even if Steam itself - and more importantly, the data files we need such asconfig.vdf
- do not. So because the path tocompatibilitytools.d
can exist, we might mark this Steam installation as valid by adding it toavailable_dirs
. But then if we try to use this Steam directory, we'll be unable to read the files we need such asconfig.vdf
because the other data files are missing.We already encountered this problem in a different form a while back, which we solved in #231. However that issue was about selecting the correct Steam installation path, assuming that at least one of them existed. It's a similar idea as the problem was with selecting a Steam installation path that existed without the data files we needed, but this time we need to solve for a case when the path exists, but no Steam installation exists!
In
constants.py
we do a check when selecting our_STEAM_ROOT
path to make sure we only pick one that hasconfig.vdf
andlibraryfolders.vdf
. However we will always default to_POSSIBLE_STEAM_ROOTS[0]
, which at time of writing, is~/.local/share/Steam
. This means even if our check for these VDF files fails inconstants.py
, we'll always default to using~/.local/share/Steam
. And in the case of #353, this path exists but is not valid. In other words, all the directories exist, but we don't have the VDF files we're looking for. Then, inPOSSIBLE_INSTALL_LOCATIONS
, we set our Steam path tof'{_STEAM_ROOT}/compatibilitytools.d/'
. So when we get toutil#available_install_directories
, we only check if this path exists, and if it does, we just mark it as available.Solution
The fix I came up with was to make a util function,
is_valid_launcher_installation
. This takes a dictionary fromPOSSIBLE_INSTALL_LOCATIONS
and then based on the launcher'sdisplay_name
, performs any launcher-specific checks to see if the installation is actually valid, and if no launcher specific checks are required we just fall back to theos.path.exists
check. Right now we only check "regular" Steam to solve for the problem outlined above, but this could be expanded in future.For Steam, I created a Steam utility function creatively named
is_valid_steam_install
. This takes theinstall_path
and performs checks to make sureconfig.vdf
andlibraryfolders.vdf
actually exist. This is the same check we perform inconstants.py
, actually pretty much entirely copied from there, just with the Steam path variable name changed.In short, the previous flow was:
POSSIBLE_INSTALL_LOCATIONS
,install_path
existsNow, the flow has become a bit more involved:
POSSIBLE_INSTALL_LOCATIONS
:is_valid_launcher_installation
to determine if the installation path is valid.is_valid_launcher_installation
,'Steam'
, useis_valid_steam_install
to determine if we have a valid Steam installation by passing itinstall_path
.is_valid_steam_install
.install_dir/..
(since for Steam,install_dir
always points tocompatibilitytools.d
, so we go one directory up to get the base Steam path to check against).config.vdf
andlibraryfolders.vdf
), otherwise return False.os.path.exists(install_path)
as we have no other launcher-specific logic for now.install_path
toavailable_dirs
ifis_valid_launcher_installation
is True.Concerns
I don't know if there's a better way to do this; could (or rather, should)
constants.py
re-use theis_valid_steam_install
function fromsteamutil.py
? I'm not sure if it's a good idea to import a function like this into the constant file, perhaps we'll just have to live with the duplication for now.Other than that, this was just a solution I came up with some days ago at the coffee shop. I tried to make this flexible and extensible, in case we want to extend this to the other Steam flavours or other launchers.
is_valid_steam_install
should work for Steam Flatpak and Steam Snap, but I don't have those installed to test that out, which is why I left them out of this PR. Andis_valid_launcher_installation
should be easily extensible to other launchers as well, possibly with the use of otherlutrisutil
orheroicutil
methods. Wrapping all of the launcher-specific checks inis_valid_launcher_installation
helps keep the check inavailable_install_directories
cleaner imo, and further wrapping the launcher-specific logic in helper methods would keepis_valid_launcher_installation
clean too.But perhaps this is unnecessary decomposition / over-engineering for this problem. I don't know if this is the best approach to take, so please let me know if you'd prefer a different approach! There is possibly opportunity to do stricter checks for the VDF files we need elsewhere in the code, so that we don't try to use them if they don't exist, but I don't necessarily think these approaches are mutually exclusive.
Future Work
In my tests, this didn't break detection of my Steam installation, and if the
steam_path
given tois_valid_steam_install
is invalid, then Steam will not be added to the dropdown. In this case it gracefully defaults to Lutris, but if all paths given tois_valid_launcher_installation
are invalid, we still gracefully handle this by showing the main menu but with an empty combobox and blank list, but in future there is an opportunity for some slightly improved UX here I think (not that I think many users will run into this, I think it's unlikely a user will install or use ProtonUp-Qt with no available launchers installed).Other launchers could be handled in follow-up PRs, I'm not sure if this issue affects them as much but it is not impossible I don't think. I think we're just a bit better at making sure these files exist for other launchers elsewhere, or rather, there is less dependence on these files (from my memory at least) than there is for Steam.
Took me a while to finally get around to this after my comment on the linked issue, but this was the approach I had in mind back then. I hope I explained the problem as I understand it and the solution I came up with well enough. All feedback is welcome on this! There was a lack of coffee while I was working on this but hopefully it's not a total disaster :-)
Thanks!